{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ab7504e5",
   "metadata": {},
   "source": [
    "# 数据转换"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b79d16d",
   "metadata": {},
   "source": [
    "## 日期格式"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af6b9ab3",
   "metadata": {},
   "source": [
    "|类型|标量class|数组class|pandas数据类型|\n",
    "|:--:|:--:|:--:|:--:|\n",
    "|Datetime|Timestamp | DatetimeIndex | Datetime64|\n",
    "|Timedeltas | Timedelta | TimedeltaIndex | timedelta64|\n",
    "|Timespans | Period | PeriodIndex | period|\n",
    "|Dateoffsets | Dateoffset| None | None|\n",
    "\n",
    "- datetime64是numpy处理时间相关内容的数据类型，可以对时间类型数据做灵活处理，同时还可以⽀持各种时间单位的操作\n",
    "- timedelta 时间差: differences between two datetimes\n",
    "- 如果dtype为Datetime64, 可以使用dt方法做向量化处理\n",
    "- 以DatetimeIndex为index的Series，**为TimeSries时间序列**（创建时刻数据TimeSeries时间序列）\n",
    "- DateOffset() 实现时间戳位移\n",
    "\n",
    "\n",
    "- Pyhton处理时间序列的常⽤库有time，datetime，dateutil\n",
    "\n",
    "- NOTE\n",
    "\n",
    "    - numpy.datetime64本质上是一个瘦包装器int64.它几乎没有日期/时间特定功能.\n",
    "\n",
    "    - pd.Timestamp是numpy.datetime64的包装器.它由相同的int64值支持,但支持整个datetime.datetime接口,以及有用的特定于熊猫的功能.\n",
    "\n",
    "    - 这两者的数组内表示是相同的，是一个连续的int64数组. pd.Timestamp是一个标量框,可以更轻松地处理单个值."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b74dc4f4",
   "metadata": {},
   "source": [
    "- 常见的时间单位\n",
    "    - Y: Year\n",
    "    - M: Month\n",
    "    - W: Week\n",
    "    -  D: Day\n",
    "    - h: Hour\n",
    "    - m: Minute\n",
    "    - s: Second\n",
    "    - ms: millisecond\n",
    "    - us: micorosecond\n",
    "    - ns: nanosecond\n",
    "    - ps: picosecond\n",
    "    - fs: femtosecond\n",
    "    - as: attosecond"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5aafee58",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e900953",
   "metadata": {},
   "source": [
    "### strptime()  & strftime()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "dc451635",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-04-21 11:56:31.596167\n",
      "2022年04月21日, 11:56:31\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2022, 4, 21, 0, 0)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "# strftime() : 时间，日期的格式化处理，返回以字符串表示的时间\n",
    "# strptime() : 根据指定的格式把一个时间字符串转化为日期格式\n",
    "\n",
    "now = datetime.now()\n",
    "print(now)\n",
    "now_format = now.strftime('%Y年%m月%d日, %H:%M:%S')\n",
    "print(now_format)\n",
    "\n",
    "date = '2022-04-21'\n",
    "datetime.strptime(date, '%Y-%m-%d')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e64939e",
   "metadata": {},
   "source": [
    "### python中时间日期格式化符号：\n",
    "\n",
    "- %y 两位数的年份表示（00-99）\n",
    "- %Y 四位数的年份表示（000-9999）\n",
    "- %m 月份（01-12）\n",
    "- %d 月内中的一天（0-31）\n",
    "- %H 24小时制小时数（0-23）\n",
    "- %I 12小时制小时数（01-12）\n",
    "- %M 分钟数（00=59）\n",
    "- %S 秒（00-59）\n",
    "- %a 本地简化星期名称\n",
    "- %A 本地完整星期名称\n",
    "- %b 本地简化的月份名称\n",
    "- %B 本地完整的月份名称\n",
    "- %c 本地相应的日期表示和时间表示\n",
    "- %j 年内的一天（001-366）\n",
    "- %p 本地A.M.或P.M.的等价符\n",
    "- %U 一年中的星期数（00-53）星期天为星期的开始\n",
    "- %w 星期（0-6），星期天为星期的开始\n",
    "- %W 一年中的星期数（00-53）星期一为星期的开始\n",
    "- %x 本地相应的日期表示\n",
    "- %X 本地相应的时间表示\n",
    "- %Z 当前时区的名称\n",
    "- %% %号本身"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c20cbd6",
   "metadata": {},
   "source": [
    "### datetime64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ec4f4554",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-02\n",
      "<class 'numpy.datetime64'>\n",
      "1980\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(['2022-02-02'], dtype='datetime64[D]')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date = np.datetime64('2022-02-02','M')  #'M': 时间单位\n",
    "print(date)\n",
    "print(type(date))\n",
    "\n",
    "date = np.datetime64(10, 'Y')  \n",
    "# A datetime stored as a 64-bit integer, counting from ``1970-01-01T00:00:00``.\n",
    "print(date)\n",
    "\n",
    "date = np.array(['2022-02-02'], dtype = np.datetime64)\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c711439f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['2022-02-02', '2022-02-03', '2022-02-04', '2022-02-05',\n",
       "       '2022-02-06', '2022-02-07', '2022-02-08', '2022-02-09',\n",
       "       '2022-02-10', '2022-02-11'], dtype='datetime64[D]')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 可以做向量化运算\n",
    "date+np.arange(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a3cdf7f",
   "metadata": {},
   "source": [
    "### Timestamp & DatetimeIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "de9fae41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-03-09 00:00:00\n",
      "2022-03-01 00:00:00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Timestamp('2022-03-05 00:00:00')"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date = pd.Timestamp('2022-03-09')\n",
    "print(date)\n",
    "\n",
    "date = pd.Timestamp(2022, 3, 1)\n",
    "print(date)\n",
    "\n",
    "pd.Timestamp(\"5th of Mar, 2022\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7c22e93a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2022-04-21 00:00:00')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# date = pd.to_datetime(\"5th of Mar, 2022\")\n",
    "date = pd.Timestamp(2022,4,21)\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0bb1ae51",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2022, 4, 21, 111)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date.year, date.month, date.day, date.day_of_year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9a425839",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__add__',\n",
       " '__array_priority__',\n",
       " '__class__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__pyx_vtable__',\n",
       " '__radd__',\n",
       " '__reduce__',\n",
       " '__reduce_cython__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__rsub__',\n",
       " '__setattr__',\n",
       " '__setstate__',\n",
       " '__setstate_cython__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__sub__',\n",
       " '__subclasshook__',\n",
       " '__weakref__',\n",
       " '_date_repr',\n",
       " '_freq',\n",
       " '_freqstr',\n",
       " '_repr_base',\n",
       " '_round',\n",
       " '_set_freq',\n",
       " '_short_repr',\n",
       " '_time_repr',\n",
       " 'asm8',\n",
       " 'astimezone',\n",
       " 'ceil',\n",
       " 'combine',\n",
       " 'ctime',\n",
       " 'date',\n",
       " 'day',\n",
       " 'day_name',\n",
       " 'day_of_week',\n",
       " 'day_of_year',\n",
       " 'dayofweek',\n",
       " 'dayofyear',\n",
       " 'days_in_month',\n",
       " 'daysinmonth',\n",
       " 'dst',\n",
       " 'floor',\n",
       " 'fold',\n",
       " 'freq',\n",
       " 'freqstr',\n",
       " 'fromisocalendar',\n",
       " 'fromisoformat',\n",
       " 'fromordinal',\n",
       " 'fromtimestamp',\n",
       " 'hour',\n",
       " 'is_leap_year',\n",
       " 'is_month_end',\n",
       " 'is_month_start',\n",
       " 'is_quarter_end',\n",
       " 'is_quarter_start',\n",
       " 'is_year_end',\n",
       " 'is_year_start',\n",
       " 'isocalendar',\n",
       " 'isoformat',\n",
       " 'isoweekday',\n",
       " 'max',\n",
       " 'microsecond',\n",
       " 'min',\n",
       " 'minute',\n",
       " 'month',\n",
       " 'month_name',\n",
       " 'nanosecond',\n",
       " 'normalize',\n",
       " 'now',\n",
       " 'quarter',\n",
       " 'replace',\n",
       " 'resolution',\n",
       " 'round',\n",
       " 'second',\n",
       " 'strftime',\n",
       " 'strptime',\n",
       " 'time',\n",
       " 'timestamp',\n",
       " 'timetuple',\n",
       " 'timetz',\n",
       " 'to_datetime64',\n",
       " 'to_julian_date',\n",
       " 'to_numpy',\n",
       " 'to_period',\n",
       " 'to_pydatetime',\n",
       " 'today',\n",
       " 'toordinal',\n",
       " 'tz',\n",
       " 'tz_convert',\n",
       " 'tz_localize',\n",
       " 'tzinfo',\n",
       " 'tzname',\n",
       " 'utcfromtimestamp',\n",
       " 'utcnow',\n",
       " 'utcoffset',\n",
       " 'utctimetuple',\n",
       " 'value',\n",
       " 'week',\n",
       " 'weekday',\n",
       " 'weekofyear',\n",
       " 'year']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(date)  # 查看属性和方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "379b74c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date.weekofyear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8635830c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2022-03-05', '2022-03-12', '2022-03-19', '2022-03-26',\n",
       "               '2022-04-02', '2022-04-09', '2022-04-16', '2022-04-23',\n",
       "               '2022-04-30', '2022-05-07'],\n",
       "              dtype='datetime64[ns]', freq=None)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 可以做向量化运算\n",
    "date + pd.to_timedelta(np.arange(10),'D')  # 类型转化为DatetimeIndex\n",
    "date + pd.to_timedelta(np.arange(10),'W')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "id": "4ec91ee7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatetimeIndex(['2022-02-02', '2021-06-05', '2020-09-01', '2021-09-02',\n",
      "               '2019-09-01', '2022-05-06', '2022-01-01'],\n",
      "              dtype='datetime64[ns]', freq=None)\n"
     ]
    }
   ],
   "source": [
    "# 构建时间序列 DatetimeIndex\n",
    "\n",
    "from datetime import datetime\n",
    "date = pd.to_datetime([datetime(2022,2,2), '5th of June, 2021', '2020-Sep-1',\n",
    " \"09-02-2021\", '20190901', '2022-05-06','2022-1-1'])\n",
    "print(date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "53920f6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2022-04-21 00:00:00')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 数值型不能直接转换为日期格式\n",
    "pd.to_datetime(20220421)\n",
    "\n",
    "pd.to_datetime(20220421, format='%Y%m%d')\n",
    "\n",
    "pd.to_datetime(str(20220421))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0a32ba8",
   "metadata": {},
   "source": [
    "### Period & PeriodIndex \n",
    "\n",
    "- Period 时间段表示时间轴上的某一区间\n",
    "- Period() 函数后面通常有两个参数，第二个 freq 参数决定时间段的分割长度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "07ad009e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Period('2022-01', 'M')"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date = pd.Period('2022-1-1', freq = 'M')\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0daf3083",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PeriodIndex(['2022-01-01', '2022-01-02', '2022-01-03'], dtype='period[D]')\n"
     ]
    }
   ],
   "source": [
    "# DatetimeIndex 类型通过 pd.to_period和⼀个频率代码可以转换成PeriodIndex类型\n",
    "date = pd.to_datetime(['2022-1-1', '2022-1-2','2022-1-3'])\n",
    "d = date.to_period('D')\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1434aada",
   "metadata": {},
   "source": [
    "### Timedelta & TimedeltaIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0a6aa50a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2022-01-20 11:52:55.594820')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "date = pd.Timestamp('2022-01-01')\n",
    "datediff = datetime.now() - date \n",
    "datediff\n",
    "\n",
    "date + datediff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "965849b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TimedeltaIndex([ '-79 days +11:47:11.469416', '-321 days +11:47:11.469416',\n",
      "                '-598 days +11:47:11.469416', '-232 days +11:47:11.469416',\n",
      "                '-964 days +11:47:11.469416',    '14 days 11:47:11.469416',\n",
      "                '-111 days +11:47:11.469416'],\n",
      "               dtype='timedelta64[ns]', freq=None)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([-0.21495005, -0.87752355, -1.63592379, -0.63384983, -2.63799775,\n",
       "        0.0396753 , -0.30256308])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# DatetimeIndex相减，返回的是TimedeltaIndex类型\n",
    "date = pd.to_datetime([datetime(2022,2,2), '5th of June, 2021', '2020-Sep-1',\n",
    " \"09-02-2021\", '20190901', '2022-05-06','2022-1-1'])\n",
    "date_diff = date - datetime.now()\n",
    "print(date_diff)\n",
    "\n",
    "date_diff.days\n",
    "# print(date_diff.day)\n",
    "\n",
    "# 转化为年\n",
    "date_diff/pd.Timedelta('1y')   \n",
    "# 或者\n",
    "date_diff.values/pd.Timedelta('1Y')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563c8fdf",
   "metadata": {},
   "source": [
    "### xxx_range类函数\n",
    "\n",
    "- pandas提供了可以规律产⽣时间序列的函数, 此类函数的使⽤和range类似，pandas提供了三个函数：\n",
    "    - pd.date_range： 可以处理时间戳\n",
    "    - pd.period_range：可以处理周期\n",
    "    - pd.timedelta_range：可以处理时间间隔"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7cb5de14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatetimeIndex(['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04',\n",
      "               '2021-01-05', '2021-01-06', '2021-01-07', '2021-01-08',\n",
      "               '2021-01-09', '2021-01-10',\n",
      "               ...\n",
      "               '2022-04-22', '2022-04-23', '2022-04-24', '2022-04-25',\n",
      "               '2022-04-26', '2022-04-27', '2022-04-28', '2022-04-29',\n",
      "               '2022-04-30', '2022-05-01'],\n",
      "              dtype='datetime64[ns]', length=486, freq='D')\n",
      "DatetimeIndex(['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04',\n",
      "               '2021-01-05', '2021-01-06', '2021-01-07', '2021-01-08',\n",
      "               '2021-01-09', '2021-01-10'],\n",
      "              dtype='datetime64[ns]', freq='D')\n",
      "DatetimeIndex(['2021-01-01', '2021-01-11', '2021-01-21', '2021-01-31',\n",
      "               '2021-02-10', '2021-02-20', '2021-03-02', '2021-03-12',\n",
      "               '2021-03-22', '2021-04-01'],\n",
      "              dtype='datetime64[ns]', freq='10D')\n",
      "DatetimeIndex(['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04',\n",
      "               '2021-01-05', '2021-01-06', '2021-01-07', '2021-01-08',\n",
      "               '2021-01-09', '2021-01-10',\n",
      "               ...\n",
      "               '2021-05-20', '2021-05-21', '2021-05-22', '2021-05-23',\n",
      "               '2021-05-24', '2021-05-25', '2021-05-26', '2021-05-27',\n",
      "               '2021-05-28', '2021-05-29'],\n",
      "              dtype='datetime64[ns]', length=149, freq='D')\n"
     ]
    }
   ],
   "source": [
    "date = pd.date_range(start='20210101',end='20220501')\n",
    "print(date)\n",
    "\n",
    "# start  开始日期\n",
    "# end    结束日期 \n",
    "# periods 时间个数\n",
    "# freq    时间间隔 \n",
    "\n",
    "date = pd.date_range(start='20210101',periods=10)\n",
    "print(date)\n",
    " \n",
    "date = pd.date_range(start='20210101',periods=10,freq='10D')\n",
    "print(date)\n",
    "\n",
    "date = pd.date_range(start='20210101',end='20210530',closed='left')\n",
    "print(date)\n",
    "# closed ='left' 包含左边, 不包含右边\n",
    "# closed = 'right'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "399e0ea1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PeriodIndex(['2018-01', '2018-02', '2018-03', '2018-04', '2018-05'], dtype='period[M]')\n"
     ]
    }
   ],
   "source": [
    "# period_range\n",
    "date = pd.period_range(\"2018-01-01\", periods=5, freq=\"M\")\n",
    "print(date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bd20e55f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TimedeltaIndex(['0 days 00:00:00', '0 days 01:00:00', '0 days 02:00:00',\n",
      "                '0 days 03:00:00', '0 days 04:00:00'],\n",
      "               dtype='timedelta64[ns]', freq='H')\n"
     ]
    }
   ],
   "source": [
    "# timedelta_range\n",
    "date = pd.timedelta_range(0, periods=5, freq=\"H\")\n",
    "print(date)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "305266c3",
   "metadata": {},
   "source": [
    "### 时间序列在dataframe作用 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd0a63e",
   "metadata": {},
   "source": [
    "#### 查找数据 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2cb7b2b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2022-03-01 07:00:00    5\n",
       "2022-03-01 08:00:00    2\n",
       "2022-03-01 09:00:00    8\n",
       "2022-03-01 10:00:00    2\n",
       "2022-03-01 11:00:00    7\n",
       "2022-03-01 12:00:00    6\n",
       "2022-03-01 13:00:00    7\n",
       "2022-03-01 14:00:00    5\n",
       "2022-03-01 15:00:00    2\n",
       "2022-03-01 16:00:00    3\n",
       "2022-03-01 17:00:00    2\n",
       "2022-03-01 18:00:00    9\n",
       "2022-03-01 19:00:00    8\n",
       "dtype: int32"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "date = pd.to_datetime('2022-3-1') + pd.to_timedelta(np.arange(10),'W')\n",
    "df = pd.Series(np.random.randint(0,10,size=10),index=date)\n",
    "df  # timeseries data\n",
    "\n",
    "# 查找\n",
    "df['2022-4']\n",
    "df['2022-4':]\n",
    "\n",
    "#truncate 满足条件的删掉 \n",
    "after = df.truncate(after='2022-03-19') \n",
    "after\n",
    "\n",
    "before = df.truncate(before='2022-03-19') \n",
    "before\n",
    "\n",
    "df.truncate(before='2022-03-19', after = '2022-04-09') \n",
    "\n",
    "\n",
    "# 查找满足要求时间段 \n",
    "date = pd.to_datetime('2022-3-1') + pd.to_timedelta(np.arange(20),'h')\n",
    "df = pd.Series(np.random.randint(0,10,size=20),index=date)\n",
    "df  # timeseries data\n",
    "df.between_time(\"7:00\",\"20:00\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045d73ad",
   "metadata": {},
   "source": [
    "#### 移位日期"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "90a47eb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-01-01    7\n",
      "2022-01-02    3\n",
      "2022-01-03    6\n",
      "2022-01-04    5\n",
      "2022-01-05    5\n",
      "2022-01-06    4\n",
      "2022-01-07    3\n",
      "2022-01-08    9\n",
      "2022-01-09    1\n",
      "2022-01-10    0\n",
      "Freq: D, dtype: int32\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2022-01-02    7\n",
       "2022-01-03    3\n",
       "2022-01-04    6\n",
       "2022-01-05    5\n",
       "2022-01-06    5\n",
       "2022-01-07    4\n",
       "2022-01-08    3\n",
       "2022-01-09    9\n",
       "2022-01-10    1\n",
       "2022-01-11    0\n",
       "Freq: D, dtype: int32"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = pd.date_range(start='20220101',periods=10) \n",
    "df = pd.Series(np.random.randint(0,10,size=10),index=index)\n",
    "print(df)\n",
    "\n",
    "# 将元素列向下移动一条。\n",
    "df.shift(1)  #,fill_value=0)\n",
    "\n",
    "# 也可以移动索引，元素不变，需要加上 freq 参数。将索引列向下移动一条\n",
    "df.shift(1,freq = 'D')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f74e68",
   "metadata": {},
   "source": [
    "#### 重采样\n",
    "\n",
    "- 可以理解为改变时间索引的个数，通过增大或减小相邻索引的时间间隔以达到减小或增加索引数量的效果\n",
    "- 使用 resample() 函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "89eaa1dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01    18\n",
      "2018-10-02    24\n",
      "2018-10-03     1\n",
      "2018-10-04    33\n",
      "2018-10-05     3\n",
      "2018-10-06     8\n",
      "2018-10-07    17\n",
      "2018-10-08    42\n",
      "2018-10-09    47\n",
      "2018-10-10    17\n",
      "Freq: D, dtype: int32\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2018-10-01 00:00:00    18\n",
       "2018-10-01 12:00:00    18\n",
       "2018-10-02 00:00:00    24\n",
       "2018-10-02 12:00:00    24\n",
       "2018-10-03 00:00:00     1\n",
       "2018-10-03 12:00:00     1\n",
       "2018-10-04 00:00:00    33\n",
       "2018-10-04 12:00:00    33\n",
       "2018-10-05 00:00:00     3\n",
       "2018-10-05 12:00:00     3\n",
       "2018-10-06 00:00:00     8\n",
       "2018-10-06 12:00:00     8\n",
       "2018-10-07 00:00:00    17\n",
       "2018-10-07 12:00:00    17\n",
       "2018-10-08 00:00:00    42\n",
       "2018-10-08 12:00:00    42\n",
       "2018-10-09 00:00:00    47\n",
       "2018-10-09 12:00:00    47\n",
       "2018-10-10 00:00:00    17\n",
       "Freq: 12H, dtype: int32"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rng = pd.date_range('10/1/2018', periods=10, freq='D')\n",
    "ts = pd.Series(np.random.randint(0, 50, len(rng)), index=rng)\n",
    "print(ts)\n",
    "\n",
    "# 下采样，增加时间间隔\n",
    "ts.resample('W').sum()\n",
    "ts.resample('W').mean()\n",
    "\n",
    "ts.resample('W').ohlc()  #使用 ohlc() 函数对所用未被采样值进行统计\n",
    "\n",
    "# 上采样，减少时间间隔\n",
    "ts.resample('12H').asfreq()\n",
    "# ffill() 函数可以将新增的索引值以相邻的前一条索引值进行填充\n",
    "ts.resample('12H').ffill()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25b494e6",
   "metadata": {},
   "source": [
    "### 时间的算术方法\n",
    "\n",
    "- check pandas.tseries.offsets 模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "051cc8ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2022-02-03 00:00:00')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = pd.DateOffset(months = 1, days=2)\n",
    "a = pd.to_datetime('2022-1-1')\n",
    "a+b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1059329b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2021-12-06 00:00:00')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pandas.tseries.offsets import BDay\n",
    "\n",
    "a -  20*BDay()  #工作日"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab53043",
   "metadata": {},
   "source": [
    "- 一些方法\n",
    "  \n",
    "data['日期'] =data_2019['交易时间'].dt.date\n",
    "\n",
    "data['时间'] =data_2019['交易时间'].dt.time\n",
    "\n",
    "data['年'] = data_2019['交易时间'].dt.year\n",
    "\n",
    "data['季节'] = data_2019['交易时间'].dt.quarter\n",
    "\n",
    "data['月'] = data_2019['交易时间'].dt.month\n",
    "\n",
    "data['周']=data_2019['交易时间'].dt.week\n",
    "\n",
    "data['日'] = data_2019['交易时间'].dt.day\n",
    "\n",
    "data['小时'] =data_2019['交易时间'].dt.hour\n",
    "\n",
    "data['分钟'] =data_2019['交易时间'].dt.minute\n",
    "\n",
    "data['秒'] = data_2019['交易时间'].dt.second\n",
    "\n",
    "data['一年第几天'] =data_2019['交易时间'].dt.dayofyear\n",
    "\n",
    "data['一年第几周'] = data_2019['交易时间'].dt.weekofyear\n",
    "\n",
    "data['一周第几天'] = data_2019['交易时间'].dt.dayofweek\n",
    "\n",
    "data['一个月含有多少天'] = data_2019['交易时间'].dt.days_in_month\n",
    "\n",
    "data['星期名称'] =data_2019['交易时间'].dt.weekday_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e9b6d0",
   "metadata": {},
   "source": [
    "### 练习\n",
    "\n",
    "- 需求：计算被访人在被访问时点的年龄"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa1fed6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "data_path = r'C:\\Users\\illus\\OneDrive\\A-Teaching\\Data-Analysis\\raw-data' \n",
    "os.chdir(data_path)\n",
    "#os.listdir()\n",
    "df = pd.read_csv('cfps2018_famconf_demo.csv', dtype = {'pid':int})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "4e1617ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fid18</th>\n",
       "      <th>fid_provcd18</th>\n",
       "      <th>fid_countyid18</th>\n",
       "      <th>fid_cid18</th>\n",
       "      <th>fid_urban18</th>\n",
       "      <th>pid</th>\n",
       "      <th>fid_base</th>\n",
       "      <th>psu</th>\n",
       "      <th>fid10</th>\n",
       "      <th>fid12</th>\n",
       "      <th>...</th>\n",
       "      <th>ads1_18</th>\n",
       "      <th>kz103_18</th>\n",
       "      <th>interrupt18</th>\n",
       "      <th>sresppid18</th>\n",
       "      <th>kz1pid18</th>\n",
       "      <th>cyear18</th>\n",
       "      <th>cmonth18</th>\n",
       "      <th>iwmode18</th>\n",
       "      <th>interviewerid18</th>\n",
       "      <th>releaseversion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100051.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>624942.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>100051501</td>\n",
       "      <td>110043.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100051502.0</td>\n",
       "      <td>100051502.0</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>761040.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100051.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>624942.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>110043107</td>\n",
       "      <td>110043.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>110043.0</td>\n",
       "      <td>110043.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100051502.0</td>\n",
       "      <td>100051502.0</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>761040.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100051.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>624942.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>100051502</td>\n",
       "      <td>110043.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100051502.0</td>\n",
       "      <td>100051502.0</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>761040.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100160.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>100160601</td>\n",
       "      <td>120009.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120009102.0</td>\n",
       "      <td>120009102.0</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>459505.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100160.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>120009102</td>\n",
       "      <td>120009.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>120009.0</td>\n",
       "      <td>120009.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120009102.0</td>\n",
       "      <td>120009102.0</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>459505.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 296 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      fid18  fid_provcd18  fid_countyid18  fid_cid18  fid_urban18        pid  \\\n",
       "0  100051.0          11.0            45.0   624942.0          1.0  100051501   \n",
       "1  100051.0          11.0            45.0   624942.0          1.0  110043107   \n",
       "2  100051.0          11.0            45.0   624942.0          1.0  100051502   \n",
       "3  100160.0          12.0            79.0       -9.0          1.0  100160601   \n",
       "4  100160.0          12.0            79.0       -9.0          1.0  120009102   \n",
       "\n",
       "   fid_base   psu     fid10     fid12  ...  ads1_18  kz103_18  interrupt18  \\\n",
       "0  110043.0  45.0      -8.0      -8.0  ...      0.0       1.0          0.0   \n",
       "1  110043.0  45.0  110043.0  110043.0  ...      0.0       1.0          0.0   \n",
       "2  110043.0  45.0      -8.0      -8.0  ...      0.0       1.0          0.0   \n",
       "3  120009.0  79.0      -8.0      -8.0  ...      0.0       1.0          0.0   \n",
       "4  120009.0  79.0  120009.0  120009.0  ...      0.0       1.0          0.0   \n",
       "\n",
       "    sresppid18     kz1pid18  cyear18  cmonth18  iwmode18  interviewerid18  \\\n",
       "0  100051502.0  100051502.0   2018.0      10.0       2.0         761040.0   \n",
       "1  100051502.0  100051502.0   2018.0      10.0       2.0         761040.0   \n",
       "2  100051502.0  100051502.0   2018.0      10.0       2.0         761040.0   \n",
       "3  120009102.0  120009102.0   2018.0       8.0       1.0         459505.0   \n",
       "4  120009102.0  120009102.0   2018.0       8.0       1.0         459505.0   \n",
       "\n",
       "   releaseversion  \n",
       "0             1.0  \n",
       "1             1.0  \n",
       "2             1.0  \n",
       "3             1.0  \n",
       "4             1.0  \n",
       "\n",
       "[5 rows x 296 columns]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "7a72caad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pid</th>\n",
       "      <th>birth_y</th>\n",
       "      <th>birth_m</th>\n",
       "      <th>inter_y</th>\n",
       "      <th>inter_m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100051501</td>\n",
       "      <td>1969</td>\n",
       "      <td>1</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110043107</td>\n",
       "      <td>1994</td>\n",
       "      <td>12</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100051502</td>\n",
       "      <td>1966</td>\n",
       "      <td>1</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100160601</td>\n",
       "      <td>1989</td>\n",
       "      <td>2</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>120009102</td>\n",
       "      <td>1991</td>\n",
       "      <td>8</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         pid  birth_y  birth_m  inter_y  inter_m\n",
       "0  100051501     1969        1   2018.0     10.0\n",
       "1  110043107     1994       12   2018.0     10.0\n",
       "2  100051502     1966        1   2018.0     10.0\n",
       "3  100160601     1989        2   2018.0      8.0\n",
       "4  120009102     1991        8   2018.0      8.0"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 出生年、月：tb1y_a_p, tb1m_a_p ; 调查年、月：cyear18, cmonth18 用于计算访问时的年龄\n",
    "df_use = df[['pid','tb1y_a_p', 'tb1m_a_p','cyear18', 'cmonth18']]\n",
    "df_use.columns = ['pid','birth_y','birth_m','inter_y','inter_m']\n",
    "df_use.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72307cb0",
   "metadata": {},
   "source": [
    "- 需要做的事情\n",
    "    - i 检查相关数据，是否有缺失值，思考如何处理\n",
    "    - ii 将生日转换为datetime格式, 计算时间差"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "5d325cd8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 10    950\n",
      " 11    869\n",
      " 9     863\n",
      " 3     838\n",
      " 5     828\n",
      " 2     819\n",
      " 4     799\n",
      " 12    796\n",
      " 7     782\n",
      " 8     772\n",
      " 1     763\n",
      " 6     720\n",
      "-9     201\n",
      "Name: birth_m, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "birth_y     51\n",
       "birth_m    201\n",
       "inter_y      0\n",
       "inter_m      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# i 检查数据，是否有缺失值，思考如何处理\n",
    "# CFPS编码方式：-8 不适用， -9 缺失，-1 不知道\n",
    "\n",
    "# 看一下出生年月\n",
    "print(df_use['birth_m'].value_counts())\n",
    "# 月份数据包含 -9，有201人，如何补充数据？随机填？ 填 mode？ 填 6月？ \n",
    "df_use['birth_y'].unique()\n",
    "# 年份数据包含 -9， -1\n",
    "sum(df_use['birth_y']<0)  # 数量为51，处理方法：剔除，不计算他们的年龄; \n",
    "\n",
    "# 思考如何填月份，查看没有月份数据的出生年份\n",
    "df_use.loc[df_use['birth_m']<0,'birth_y'].value_counts()  # 没有集中年份缺失，使用随机填月份\n",
    "\n",
    "# 总体看一下缺失的数据(调查时间应没有缺失值)\n",
    "np.sum(df_use.iloc[:,1:]<0,axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "165b7743",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 剔除出生年份缺失数据\n",
    "df_use = df_use[df_use['birth_y']>0]\n",
    "# 填补出生月份数据\n",
    "temp = sum(df_use['birth_m']<0) \n",
    "a = np.random.randint(1,13, temp)\n",
    "df_use.loc[df_use['birth_m']<0,'birth_m'] = a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c16150b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_use.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "id": "a3a2336f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ii 转换为datetime, 计算时间差\n",
    "def date_transform(tem,date):\n",
    "    col_name = tem.columns.to_list()\n",
    "    tem = df_use.rename(columns={col_name[0]:'year',col_name[1]:'month'})\n",
    "    df_use[date] = pd.to_datetime(tem[['year','month','day']],errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "id": "cf47524e",
   "metadata": {},
   "outputs": [],
   "source": [
    "date_transform(df_use[['birth_y','birth_m','day']],'birthday')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "id": "dfbe19e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "date_transform(df_use[['inter_y','inter_m','day']],'interday')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "id": "f928db60",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\IPython\\core\\interactiveshell.py:3444: FutureWarning: Units 'M', 'Y' and 'y' do not represent unambiguous timedelta values and will be removed in a future version\n",
      "  exec(code_obj, self.user_global_ns, self.user_ns)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pid</th>\n",
       "      <th>birth_y</th>\n",
       "      <th>birth_m</th>\n",
       "      <th>inter_y</th>\n",
       "      <th>inter_m</th>\n",
       "      <th>day</th>\n",
       "      <th>birthday</th>\n",
       "      <th>interday</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100051501</td>\n",
       "      <td>1969</td>\n",
       "      <td>1</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1969-01-01</td>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>49.747770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110043107</td>\n",
       "      <td>1994</td>\n",
       "      <td>12</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1994-12-01</td>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>23.833480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100051502</td>\n",
       "      <td>1966</td>\n",
       "      <td>1</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1966-01-01</td>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>52.748516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100160601</td>\n",
       "      <td>1989</td>\n",
       "      <td>2</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1989-02-01</td>\n",
       "      <td>2018-08-01</td>\n",
       "      <td>29.495472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>120009102</td>\n",
       "      <td>1991</td>\n",
       "      <td>8</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1991-08-01</td>\n",
       "      <td>2018-08-01</td>\n",
       "      <td>27.001239</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         pid birth_y  birth_m  inter_y  inter_m  day   birthday   interday  \\\n",
       "0  100051501    1969        1   2018.0     10.0    1 1969-01-01 2018-10-01   \n",
       "1  110043107    1994       12   2018.0     10.0    1 1994-12-01 2018-10-01   \n",
       "2  100051502    1966        1   2018.0     10.0    1 1966-01-01 2018-10-01   \n",
       "3  100160601    1989        2   2018.0      8.0    1 1989-02-01 2018-08-01   \n",
       "4  120009102    1991        8   2018.0      8.0    1 1991-08-01 2018-08-01   \n",
       "\n",
       "         age  \n",
       "0  49.747770  \n",
       "1  23.833480  \n",
       "2  52.748516  \n",
       "3  29.495472  \n",
       "4  27.001239  "
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_use['age'] = (df_use['interday'] - df_use['birthday'])/pd.Timedelta('1Y')\n",
    "df_use.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "fa79e9e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new = pd.merge(df,df_use[['pid','age']],on = 'pid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "id": "c9a275a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pid</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100051501</td>\n",
       "      <td>49.747770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110043107</td>\n",
       "      <td>23.833480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100051502</td>\n",
       "      <td>52.748516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100160601</td>\n",
       "      <td>29.495472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>120009102</td>\n",
       "      <td>27.001239</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         pid        age\n",
       "0  100051501  49.747770\n",
       "1  110043107  23.833480\n",
       "2  100051502  52.748516\n",
       "3  100160601  29.495472\n",
       "4  120009102  27.001239"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new[['pid','age']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c2d647e",
   "metadata": {},
   "source": [
    "## 文本格式\n",
    "\n",
    "- 对object数据类型，可以使用.str方法，进行向量化操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fb933e9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "data_path = r'../../raw-data' \n",
    "os.chdir(data_path)\n",
    "#os.listdir()\n",
    "df = pd.read_csv('titanic.csv', dtype = {'pid':int})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc45c163",
   "metadata": {},
   "source": [
    "- 字段含义：\n",
    "    - SibSp：与乘客同行的兄弟姐妹（Siblings）和配偶（Spouse）数目；\n",
    "    - Parch：与乘客同行的家长（Parents）和孩子（Children）数目；\n",
    "    - Ticket（船票号）\n",
    "    - Cabin（船舱）: 由两部分组成，仓位号和房间编号，如C88中，C和88分别对应C仓位和88号房间\n",
    "    - Embarked（港口）：乘客上船时的港口，包含三种类型：\n",
    "        - C：Cherbourg；\n",
    "        - Q：Queenstown；\n",
    "        - S：Southampton；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2d16fc65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "62ba2d7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 12 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   PassengerId  891 non-null    int64  \n",
      " 1   Survived     891 non-null    int64  \n",
      " 2   Pclass       891 non-null    int64  \n",
      " 3   Name         891 non-null    object \n",
      " 4   Sex          891 non-null    object \n",
      " 5   Age          714 non-null    float64\n",
      " 6   SibSp        891 non-null    int64  \n",
      " 7   Parch        891 non-null    int64  \n",
      " 8   Ticket       891 non-null    object \n",
      " 9   Fare         891 non-null    float64\n",
      " 10  Cabin        204 non-null    object \n",
      " 11  Embarked     889 non-null    object \n",
      "dtypes: float64(2), int64(5), object(5)\n",
      "memory usage: 83.7+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caf3e7f2",
   "metadata": {},
   "source": [
    "### python中字符串的⽅法可以直接使用\n",
    "\n",
    "- i.e. `df.Name.str.upper()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "587e3736",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算每个字符串的长度\n",
    "df.Name.str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "beeade0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        MALE\n",
       "1      FEMALE\n",
       "2      FEMALE\n",
       "3      FEMALE\n",
       "4        MALE\n",
       "        ...  \n",
       "886      MALE\n",
       "887    FEMALE\n",
       "888    FEMALE\n",
       "889      MALE\n",
       "890      MALE\n",
       "Name: Sex, Length: 891, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# upper() 转换为大写\n",
    "df.Sex.str.upper()   # 返回值是做了一个拷贝"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1f936881",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                                braund, mr. owen harris\n",
       "1      cumings, mrs. john bradley (florence briggs th...\n",
       "2                                 heikkinen, miss. laina\n",
       "3           futrelle, mrs. jacques heath (lily may peel)\n",
       "4                               allen, mr. william henry\n",
       "                             ...                        \n",
       "886                                montvila, rev. juozas\n",
       "887                         graham, miss. margaret edith\n",
       "888             johnston, miss. catherine helen \"carrie\"\n",
       "889                                behr, mr. karl howell\n",
       "890                                  dooley, mr. patrick\n",
       "Name: Name, Length: 891, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lower() 转换为小写\n",
    "df.Name.str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84b212d3",
   "metadata": {},
   "source": [
    "### 正则表达式\n",
    "\n",
    "- 在pandas中有一些正则表达式的接⼝，如以下API：\n",
    "\n",
    "    - match: 调⽤re.match， 返回bool类型内容\n",
    "    - extract: 调⽤re.match, 返回匹配的字符串组groups\n",
    "    - findall： 调⽤re.findall\n",
    "    - replace: 正则的替换模式；替换字符串\n",
    "    - contains： re.search，返回bool内容；返回表示各str是否含有指定模式的字符串\n",
    "    - count： 利⽤正则模式统计数量\n",
    "    - split：等价于 str.split， ⽀持正则\n",
    "    - rsplit： 等价于str.rsplit，⽀持正则\n",
    "        - split()对按照给定的字符对字符串中所有对应字符位置进行分别\n",
    "            - 给定划分的数量后，从左往右进行划分\n",
    "            - rsplit()，从右往左进行划分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "71515f2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "309"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# contains()  包含某个内容\n",
    "df.Name.str.contains('Mrs').sum()\n",
    "\n",
    "df.Name.str.contains('Mrs|Miss').sum()  # |:表示或： 包含Mrs 或 Miss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "935c2d3e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                                  Mr. Owen Harris\n",
       "1       Mrs. John Bradley (Florence Briggs Thayer)\n",
       "2                                      Miss. Laina\n",
       "3               Mrs. Jacques Heath (Lily May Peel)\n",
       "4                                Mr. William Henry\n",
       "                          ...                     \n",
       "886                                    Rev. Juozas\n",
       "887                           Miss. Margaret Edith\n",
       "888                 Miss. Catherine Helen \"Carrie\"\n",
       "889                                Mr. Karl Howell\n",
       "890                                    Mr. Patrick\n",
       "Name: Name, Length: 891, dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split()  # 获取每个人的姓和title\n",
    "df.Name.str.split(',').str[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f6173e36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          []\n",
       "1      [Mrs.]\n",
       "2      [Miss]\n",
       "3      [Mrs.]\n",
       "4          []\n",
       "        ...  \n",
       "886        []\n",
       "887    [Miss]\n",
       "888    [Miss]\n",
       "889        []\n",
       "890        []\n",
       "Name: Name, Length: 891, dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 替换  replace()\n",
    "# df.Sex.str.replace('m','s')\n",
    "\n",
    "# findall\n",
    "df.Name.str.findall(r'M.s.')   #.:任意字符"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2fbd46e",
   "metadata": {},
   "source": [
    "###  其他字符串的使⽤⽅法\n",
    "    \n",
    "    - get: 获取给定索引位置上的值，start=0\n",
    "        - df.str.get(i) 和 df.str[i]功能类似\n",
    "    - slice： 对元素进⾏切⽚， df.str[0:2]\n",
    "    - slice_replace: 对元素进⾏切⽚替换\n",
    "    - cat： 连接字符串\n",
    "    - repeat：重复元素\n",
    "    - normalize： 将字符串转换为Unicode规范形式\n",
    "    - pad： 在字符串的左边，右边或者两边增加空格\n",
    "    - **strip**: 删除前导和后置空格\n",
    "    - wrap：将字符串按照指定宽度换⾏\n",
    "    - join： 返回一个字符串，该字符串是给定序列中所有字符串的连接, i.e. df.Sex.str.join('-') ---> male ---> m-a-l-e  \n",
    "    - **get_dummies**: 按照分隔符提取每个元素的dummy变量，转换为one-hot编码的DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8ccba718",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      aun\n",
       "1      min\n",
       "2      ikk\n",
       "3      tre\n",
       "4      len\n",
       "      ... \n",
       "886    ntv\n",
       "887    aha\n",
       "888    hns\n",
       "889    hr,\n",
       "890    ole\n",
       "Name: Name, Length: 891, dtype: object"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对slice的使⽤和函数的直接切⽚⼀个效果\n",
    "df.Name.str.slice(2,5) #等于 df.str[2:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "de9df20a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>female</th>\n",
       "      <th>male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>891 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     female  male\n",
       "0         0     1\n",
       "1         1     0\n",
       "2         1     0\n",
       "3         1     0\n",
       "4         0     1\n",
       "..      ...   ...\n",
       "886       0     1\n",
       "887       1     0\n",
       "888       1     0\n",
       "889       0     1\n",
       "890       0     1\n",
       "\n",
       "[891 rows x 2 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 生成虚拟变量\n",
    "df.Sex.str.get_dummies()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "45d7c95b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   id   course\n",
      "0   0      B|C\n",
      "1   1    A|C|D\n",
      "2   2      B|D\n",
      "3   3    A|B|C\n",
      "4   4  A|B|C|D\n",
      "\n",
      "   A  B  C  D\n",
      "0  0  1  1  0\n",
      "1  1  0  1  1\n",
      "2  0  1  0  1\n",
      "3  1  1  1  0\n",
      "4  1  1  1  1\n"
     ]
    }
   ],
   "source": [
    "# suppose A= Micro， B= Macro， C= Stat.， D= Math\n",
    "df2 = pd.DataFrame({'id': range(5),\n",
    " 'course':[\"B|C\", \"A|C|D\", \"B|D\", \"A|B|C\", \"A|B|C|D\"]})\n",
    "print(df2)\n",
    "print()\n",
    "a = df2['course'].str.get_dummies('|')\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2081977c",
   "metadata": {},
   "source": [
    "## 高阶函数\n",
    "\n",
    "- `map()`：映射，进行数据转换\n",
    "- `apply()`，调用自定义函数对数据进行处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "af46e2cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import os\n",
    "# data_path = r'C:\\Users\\illus\\OneDrive\\A-Teaching\\Data-Analysis\\raw-data' \n",
    "# os.chdir(data_path)\n",
    "# #os.listdir()\n",
    "# df = pd.read_csv('titanic.csv', dtype = {'pid':int})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "e3e0bbf9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      男\n",
       "1      女\n",
       "2      女\n",
       "3      女\n",
       "4      男\n",
       "      ..\n",
       "886    男\n",
       "887    女\n",
       "888    女\n",
       "889    男\n",
       "890    男\n",
       "Name: Sex, Length: 891, dtype: object"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 将female 转换为 1； 将male 转换为 0\n",
    "# 方法1： np.where\n",
    "df['sex_code'] = np.where(df.Sex == 'female', 1, 0)\n",
    "# 方法2： 使用 map\n",
    "df.Sex.map({'female': 1, 'male': 0})\n",
    "# df.Sex.map(lambda x: 1 if x == 'female' else 0)\n",
    "# 方法3： 使用apply\n",
    "df.Sex.apply(lambda x: 1 if x == 'female' else 0)\n",
    "\n",
    "\n",
    "\n",
    "# 理解apply是如何操作的\n",
    "count = 0\n",
    "def sex_transform(x):\n",
    "    global count  \n",
    "    '''\n",
    "    引用全局变量，不需要golbal声明\n",
    "    修改全局变量，需要使用global声明\n",
    "    特别地，列表、字典等如果只是修改其中元素的值，可以直接使用全局变量，不需要global声明\n",
    "    '''\n",
    "    count = count + 1\n",
    "    print('id:', count)\n",
    "    print(x)\n",
    "    if x == 'female':\n",
    "        print(1)\n",
    "        return 1\n",
    "    else:\n",
    "        print(0)\n",
    "        return 0\n",
    "\n",
    "# df.Sex.apply(sex_transform)\n",
    "\n",
    "# 函数可以包含多个参数\n",
    "def sex_transform(x,y,z):\n",
    "    if x == 'female':\n",
    "        return y\n",
    "    else:\n",
    "        return z\n",
    "# df.Sex.apply(sex_transform, args = ('女','男'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "6e3ce648",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                                Br**nd, Mr. Owen Harris\n",
       "1      Cu**ngs, Mrs. John Bradley (Florence Briggs Th...\n",
       "2                                 He**kinen, Miss. Laina\n",
       "3           Fu**elle, Mrs. Jacques Heath (Lily May Peel)\n",
       "4                               Al**n, Mr. William Henry\n",
       "                             ...                        \n",
       "886                                Mo**vila, Rev. Juozas\n",
       "887                         Gr**am, Miss. Margaret Edith\n",
       "888             Jo**ston, Miss. Catherine Helen \"Carrie\"\n",
       "889                                Be**, Mr. Karl Howell\n",
       "890                                  Do**ey, Mr. Patrick\n",
       "Name: Name, Length: 891, dtype: object"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 数据脱敏处理\n",
    "df.Name.apply(lambda x: x.replace(x[2:4], '**'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "0d393871",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      22.0\n",
       "1      39.0\n",
       "2      27.0\n",
       "3      36.0\n",
       "4      35.0\n",
       "       ... \n",
       "886    27.0\n",
       "887    20.0\n",
       "888     NaN\n",
       "889    27.0\n",
       "890    32.0\n",
       "Length: 891, dtype: float64"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对dataframe整体操作\n",
    "df.dtypes\n",
    "df[['Survived','Age']].apply(np.mean, axis = 0)   # axis = 0, 每次分配一列 （穿过列）\n",
    "\n",
    "df[['Survived','Age']].apply(lambda x: sum(x), axis = 1)  # axis =1, 每次分配一行 （穿过行）"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
